The Role of the Trainer in Reinforcement Learning

In this paper we propose a three-stage incremental approach to the development of autonomous agents. We discuss some issues about the characteristics which differentiate reinforcement programs (RPs), and define the trainer as a particular kind of RP. We present a set of results obtained running experiments with a trainer which provides guidance to the AutonoMouse, our mouse-sized autonomous robot. 1 THE THREE STAGES OF THE DEVELOPMENTAL APPROACH Reinforcement learning (RL) problems are those problems in which an agent has the goal of learning how to maximize a scalar return which is functionally related to the agent's actions. Among the most studied and best known algorithms used by researchers in trying to solve RL problems there are Q-learning (Watkins, 1989), the adaptive heuristic critic (Barto, Sutton and Watkins, 1990), and the learning classifier system (Booker, Goldberg, Holland, 1989), which was used as the learning paradigm in the experiments of this paper. An issue we have been interested in recently (Dorigo, 1992; Dorigo and Colombetti, 1994) is the role that a trainer can have in helping a reinforcement learning system to develop into a working control system. In this paper we propose the three developmental phases (3DP) model, a methodology to develop a learning agent which is based on the idea that an agent should go through different stages of development during its life; we call these phases the baby phase, the young phase, and the adult phase (Fig. 1). In the baby phase the agent starts from a situation in which it has no a priori knowledge about how to act in the world. In this phase it is helped by a trainer which gives a positive or negative reward after each action performed by the agent. This phase lasts until the agent reaches a performance level defined by the designer of the system. In the young phase the trainer is disconnected and the learner goes on learning using solely environmental reinforcements. In this phase the learning agent uses delayed reinforcements to refine the knowledge acquired during the training phase. In principle, the learning agent could by-pass the baby phase, but this would make learning much harder, and therefore slower. As in the preceding case, this phase lasts until the agent reaches a designer defined performance level, which is higher than in the baby stage. In the adult phase the learning algorithm is switched off and the agent continues to perform using the acquired knowledge. The reason to switch off the learning algorithm is that in this way the computational burden d i m i n i s h e s , a s t h e a g e n t Monitoring Trainer Baby Learning Agent